1. Field of the Invention
The present invention relates generally to solid state sensors useable in time-of-flight (TOF) and/or imaging applications, and more specifically to systems and methods to enhance dynamic range for such sensors, including improving detection from weak signals returned by distant target objects.
2. Description of Related Art
Solid state sensors used in time-of-flight (TOF) applications are known in the art. Exemplary such TOF sensors are described in U.S. Pat. No. 6,323,942 to Bamji and entitled CMOS COMPATIBLE 3-D IMAGE SENSOR, the entire contents of which is incorporated herein by this reference. However before discussing TOF systems such as described in the '942 patent, it may be helpful to first consider earlier prior art systems that relied upon acquisition of luminosity data.
As used herein, sensors (e.g., cameras) are a two-dimensional array of pixel diode detectors formed on an integrated circuit (IC) substrate. The sensors output detection signals proportional to incoming optical energy falling on the array, which energy typically has been reflected from a target object some distance Z away.
Electronic circuits that provide a measure of distance from the circuit to an object are known in the art, and may be exemplified by system 10 FIG. 1. In the generalized system of FIG. 1, imaging circuitry within system 10 is used to approximate the distance (e.g., Z1, Z2, Z3) to an object 20. Typically system 10 will include a light source 30 whose light output is focused by a lens 40 and directed toward the object to be imaged, here object 20. Other prior art systems do not provide an active light source 30 and instead rely upon and indeed require ambient light reflected by the object of interest.
Various fractions of the light from source 30 may be reflected by surface portions of object 20, and is focused by a lens 50. This return light falls upon various detector devices 60, e.g., photodiodes or the like, in an array on an integrated circuit (IC) 70. Devices 60 produce a rendering of the luminosity of an object (e.g., 10) in the scene from which distance data is to be inferred. In some applications devices 60 might be charge coupled devices (CCDs) or even arrays of CMOS devices. CCDs typically are configured in a so-called bucket-brigade whereby light-detected charge by a first CCD is serial-coupled to an adjacent CCD, whose output in turn is coupled to a third CCD, and so on. This bucket-brigade configuration precludes fabricating processing circuitry on the same IC containing the CCD array. Further, CCDs provide a serial readout as opposed to a random readout. For example, if a CCD range finder system were used in a digital zoom lens application, even though most of the relevant data would be provided by a few of the CCDs in the array, it would nonetheless be necessary to readout the entire array to gain access to the relevant data, a time consuming process.
In FIG. 1 the upper portion of object 20 is intentionally shown more distant (Z3>Z3>Z1) and less reflective than the lower portion. In an range finder autofocus camera environment, devices 60 approximate average distance from the camera (e.g., from Z=0) to object 10 by examining relative luminosity data obtained from the object. Thus, in FIG. 1, the upper portion of object 20 is darker than the lower portion, and presumably is more distant from system 10 than the lower portion. In a more complicated scene, focal distance to an object or subject standing against a background would be approximated by distinguishing the subject from the background by a change in luminosity. In a range finding binocular application, the field of view is sufficiently small such that all objects in focus are at substantially the same distance. In the various applications, circuits 80, 90, 100 within system 10 would assist in this signal processing. As noted, if IC 70 includes CCDs 60, other processing circuitry such as 80, 90, 100 are formed off-chip.
Unfortunately, as described above, reflected luminosity data does not provide a truly accurate rendering of distance because the reflectivity of the object is unknown. A distant object surface with a shiny surface may reflect as much light (perhaps more) than a closer object surface with a dull, less reflective, finish.
Further, in very brightly or very dimly illuminated scenes, system 10 may either saturate due to insufficient dynamic range, or simply fail to output data that is useful.
A different approach is described in the '942 patent, in which a high speed time of flight (TOF) system was adopted that acquired both luminosity (intensity) data and TOF data. FIG. 2A is a generic block diagram depicting a TOF system 200, according to the '942 invention. System 200 can be fabricated on a single IC 210, requires no moving parts, and relatively few off-chip components, primarily a source of optical energy 220 and associated optical focusing system. Indeed if suitable shielding were provided, one might bond laser source 220 onto the common substrate upon which IC 210 is fabricated. Optical energy source 220 is preferably an LED or a laser that emits energy 35 with a wavelength of perhaps 800 nm, although other wavelengths could instead be used. Below 800 nm wavelength, emitted light starts to become visible and laser fabrication becomes more difficult. Above 900 nm laser efficiency drops off rapidly, and in any event, 1100 nm is the upper wavelength for a device fabricated on a silicon substrate, such as IC 210. As noted, by emitted light pulses having a specific wavelength, and by filtering out incoming light of different wavelength, system 200 is operable with or without ambient light. The ability of system 200 to function in the dark can be advantageous in certain security and military type imaging applications. As described below, circuitry within system 200 detects optical energy 35′ reflected by object 20, and determines from time of flight (TOF) the distance Z to the object.
System 200 includes an array 230 of pixel photodetectors (or photodiodes) 240, each of which has dedicated circuitry 250 to process detection charge output by the associated detector. In a typical application, array 230 might include 100×100 pixels 230, and thus include 100×100 processing circuits 250. Preferably IC 210 also includes a microprocessor or microcontroller unit 260, memory 270 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 280, and various computing and input/output (I/O) circuitry 285. Among other functions, controller unit 260 may perform distance to object and object velocity calculations. Preferably the two-dimensional array 230 of pixel sensing detectors is fabricated using standard commercial silicon technology, which advantageously permits fabricating circuits 250, 260, 270, 280, and 285 on the same IC. Understandably, the ability to fabricate such circuits on the same IC with the array of pixel detectors can shorten processing and delay times, due to shorter signal paths. Various design considerations for CMOS implementation of system 200 are set forth in the '942 patent.
FIG. 2B exemplifies circuit 250 is associated with each pixel detector 240 in FIG. 2A. Circuit 250 includes a pulse peak detector 300, a high speed counter 310, and has access to high speed clock 280, which preferably outputs a continuous train of high frequency clock pulses that are coupled to the input port of each high speed counter 310. Exemplary clock pulses might exhibit perhaps 250 ns pulse width at a fixed frequency of perhaps 2 GHz with a duty cycle of perhaps 0.1% while the pulses are being output. Each counter 310 has a port to receive a START signal (e.g., start now to count), a port to receive a STOP signal (e.g., stop counting now), and a port to receive a CONTROL signal (e.g., reset accumulated count now). The CONTROL and START signals are available from controller 260, the CLOCK signal is available from clock unit 280, and the STOP signal is available from pulse peak detector 300.
Prior art system 200 operates as follows. At time t0, microprocessor 260 commands light source 220 to emit a pulse of light of known wavelength, which passes through focus lens 290′ and travels to object 20 at the speed of light (C), 300 Km/sec. If light source 220 is sufficiently powerful, lens 290′ may be dispensed with. At the surface of the object being imaged at least some of the light may be reflected back toward system 200 to be sensed by the detector array. Lens 290 preferably focuses filtered incoming optical energy onto sensor array 230 such that each pixel in the array receives light from only one particular point (e.g., an object 20 surface point) in the field of view. The properties of light wave propagation allow an ordinary lens 290 to be used to focus the light onto the sensor array. If a lens is required to focus the emitted light, a single lens could be used for 290, 290′ if a mirror-type arrangement were used.
At or before time t0, each pixel counter 310 in array 230 receives a CONTROL signal from controller 260, which resets any count previously held in the counter. At time t0, controller 260 issues a START command to each counter, whereupon each counter begins to count and accumulate CLOCK pulses from clock 280. During the roundtrip time of flight (TOF) of a light pulse, each counter accumulates CLOCK pulses, with a larger number of accumulated clock pulses representing longer TOF, which is to say, greater distance between a light reflecting point on the imaged object and system 200.
The fundamental nature of focus lens 290 associated with system 200 is such that reflected light from a point on the surface of imaged object 20 will only fall upon the pixel in the array focused upon such point. Thus, at time t1, photon light energy reflected from the closest point on the surface of object 20 will pass through a lens/filter 290 and will fall upon the pixel detector 240 in array 230 focused upon that point. A filter associated with lens 290 ensures that only incoming light have the wavelength emitted by light source 220 falls upon the detector array unattenuated.
Assume that one particular pixel detector 240 within array 230 is focused upon a nearest surface point on object 20. The associated detector 300 will detect current that is output by the pixel detector in response to the incoming photon energy from such object point. Preferably pulse detector 300 is implemented as an amplifying peak detector that senses a small but rapid change in pixel output current or voltage. When the rapidly changing output current is sufficiently large to be detected, logic within detector 300 (e.g., an SR flip-flop) toggles to latch the output pulse, which is provided as the STOP signal to the associated counter 310. Thus, the number of counts accumulated within the associated counter 310 will be indicative of roundtrip TOF to the near surface of object 20, a calculable distance Z1 away.
Distance Z1 may be determined from the following relationship in which C is the velocity of light:Z1=C·(t1)/2  Eq. (1)
At some later time t2 photon energy will arrive at lens 290 from a somewhat more distant portion of object 20, and will fall upon array 230 and be detected by another pixel detector. Hitherto the counter associated with this other detector has continued to count CLOCK pulses starting from time t0, as indeed have all counters except for the counter that stopped counting at time t1. At time t2, the pulse detector associated with the pixel just now receiving and detecting incoming photon energy will issue a STOP command to the associated counter. The accumulated count in this counter will reflect roundtrip TOF to the intermediate point on object 20, distance Z2 away. Within IC 210, controller 260 executing software stored in memory 270 can calculate distance and velocity associated with the TOF data for each light reflecting point on the object surface. Such data, or even raw TOF data, may be exported from the IC as DATA, perhaps to be further processed off-chip. Preferably DATA is exported in a format compatible to computer processing systems.
In similar fashion, at time t3 yet another pixel detector in the array will detect sufficient just-arriving photon energy for its associated pulse detector 300 to issue a STOP command to the associated counter. The accumulated count in this counter represents TOF data for the farthest distance Z3 of object 20.
Some pixels in array 230 may of course not receive sufficient reflected light from the object point upon which they are focused. Thus, after a predetermined amount of time (that may be programmed into controller 260), the counter associated with each pixel in the sensor array will have been stopped due to pulse detection, or will be assumed to hold a count corresponding to a target at distance Z=infinity.
In a motion picture application, e.g., where system 200 is perhaps used to calculate surface distances to objects to be matted, assume that the sensor array is to operate continuously at 30 frames per second, which is to say each frame shall last less than 0.33 seconds. In this application, objects at distances greater than about 50,000 Km (e.g., Z=C 0.33 s/2) cannot be detected. In practical applications, however, system 200 will normally be used to image objects within a range of 100 m or less, and the above theoretical limit will not pose a problem. Thus, the predetermined time will be approximate 660 ns (e.g., 100 m 2/C), and any pixel not outputting a peak detection pulse after 660 ns may safely be assumed to be focused upon a target image point at Z=infinity.
With each detected reflected light pulse, the counter-calculated TOF distance value for each pixel in the array is determined and preferably stored in a frame buffer in RAM associated with unit 270. Preferably microprocessor 260 examines consecutive frames stored in RAM to identify objects in the field of view scene. Microprocessor 260 can then compute object velocity. In addition to calculating distance and velocity, the microprocessor and associated on-chip circuitry can be programmed to recognize desired image shapes. Any or all of this data (denoted DATA in FIG. 2A) can be exported from the IC to an external computer for further processing, for example via a universal serial bus.
The above example described how three pixel detectors receiving photon energies at three separate times t1, t2, t3 turn-off associated counters whose accumulated counts could be used to calculate distances Z1, Z2, Z3 to object 20. In practice, the present invention will process not three but thousands or tens of thousands of such calculations per each light pulse, depending upon the size of the array. Such processing can occur on IC chip 210, for example using microprocessor 260 to execute routines stored (or storable) in memory 280. Each of the pixel detectors in the array will have unique (x, y) axis locations on the detection array, and the count output from the high speed counter associated with each pixel detector can be uniquely identified. Thus, TOF data gathered by two-dimensional detection array 230 may be signal processed to provide distances to a three-dimensional object surface. Output from CMOS-compatible detectors 240 advantageously may be accessed in a random manner if desired, which permits outputting TOF DATA in any order.
In practical applications, sensor array 230 preferably has sufficient resolution to differentiate target distances on the order of 1 cm, which implies each pixel must be able to resolve time differences on the order of 70 ps (e.g., 21 cm/C). In terms of a CMOS-implemented system specification, high speed counters 310 must be able to resolve time to within about 100 ps, and peak pulse detectors 300 must be low-noise high speed units also able to resolve about 100 ps with a detection sensitivity on the order of perhaps a few microvolts (μV). Accurate distance measurements will require that the pulse detector response time be removed from the total elapsed time. Finally, the CLOCK signal output by circuit 280 should have a period on the order of about 100 ps.
In one embodiment of the '942 patent, each pixel detector 240 has an associated circuit 250 that includes an associated shutter and charge accumulator or integrator circuit 600, such as shown in FIG. 2C. Output from each pixel photodetector 240 is coupled to a dedicated amplifier 610 whose output is coupled via a shutter S1 to a charge accumulator or integrator, shown here symbolically as a capacitor C1. Shutter S1 can be open (current passing) or closed (current-blocking). Shutter S1 may be a high-speed electronic switch, or an opto-mechanical shutter such as a ferroelectric material whose opaqueness may be rapidly altered in response to electronic control signals.
FIG. 2D depicts timing relationship for the embodiment of FIG. 2C, according to the '942 patent. At an initial time t0, energy emitter 220 (see FIG. 2A) emits a pulse of energy 35 having pulse width PW represented by a time difference (Tep-To), and having an arbitrary energy level A. Some of the emitted energy will be reflected by target object 20 and is returned as a reflected pulse 35′. The beginning of the return pulse 35′ will arrive at system 200 at time t1, and the round trip time-of-flight (TOF) from emission of pulse 35 to beginning of return pulse 35′ is (t1−t0). Shutter S1 will remain open from time t0 until time Ts during which time the associated capacitor C1 will integrate charge. Let Vos be the signal that results when the shutter is open for substantially the time duration PW, and Vcs is the signal that results when the shutter is open continuously. As such, VosNcs represents the ratio of the integration time with the shutter operating and with the shutter open. Integration time with the shutter open is PW, and integration time with the shutter operating is Tshutter=PW·(VosNcs). Integration time (Tshutter) starts at time t1 when the return pulse is received, and ends at time Ts when the shutter is closed. As indicated by FIG. 2D,Tshutter=(Ts−t1)=PW−{TOF−(Ts−Tep)}.  Eq. (2)
It follows from these timing relationships for Tshutter that:
                    TOF        =                                                            Vcs                -                Vos                            Vcs                        ⋆            PW                    +          Ts          -                      Tep            .                                              Eq        .                                  ⁢                  (          3          )                    
Indeed, an electronic shutter may be constructed using a controlled source such as a current-controlled current source. A pixel detector will produce a current Is proportional to the number of received photons, which may be mirrored with a current-controlled current source to produce current Im, whereIm=K·Is  Eq. (4)where K is a constant. Rather than directly measure pixel detector charge, charge on capacitor C1 is measured. The shutter may be closed by turning-off the current-controlled current source such that Im=0, in which case current Is no longer affects capacitor C1.
Any incoming light photons falling upon detector 240 will produce a current that is amplified by unit 610. As long as shutter S1 is open, the amplified charge-detected pixel output from unit 610 will charge capacitor C1. Conversely, when shutter S1 is closed, no additional charge from unit 610 is received by charge accumulator or integrator C1.
Thus, for an object relatively near the pixel detector, substantially all of the light pulse generated photons from the object surface can reach the detector before the shutter closes. However, for a relatively distant object, photons resulting from the end of the light source pulse may not have sufficient time to reach the sensor before the shutter closes and hence will be discarded. By calculating the fraction of photons from the pulse that are blocked by the shutter, distance Z to the object can be computed.
Controller or processor 260 causes circuit 285 to drive light source 220 to emit a pulse of light at time t0. However at or before time t0, controller 260 causes each charge integrator C1 to discharge any charge, and then opens the associated shutter S1. In response to being illuminated with the emitted light pulse, different portions of the surface of object 20 reflect optical energy 35′ back towards system 200.
Eventually pixels focused upon the nearest surface of object 20 (e.g., at distance Z1 in FIG. 2A) begin to detect incoming photons. The current charge from these pixels is amplified by amplifier 610 and charges, via the open shutter S1, capacitor C1. As long as S1 is open C1 will continue to accumulate charge. After a while, pixels focused upon slightly further object distances begin to detect incoming photons, and begin to accumulate charge on their own accumulator C1, via their individual amplifier and shutter.
Note that integrators C1 associated with pixels focused on nearer object surfaces begin to integrate charge sooner in time than other integrators, and thus can accumulate more charge per unit time. After a time approximating the emitted light pulse width, controller 260 causes all shutters to close, preferably simultaneously. At this point, the accumulated charge on each accumulator C1 is static, or frozen. If desired, some or all of the shutters may be opened in any desired sequence, including randomly, rather than simultaneously.
Accumulated charge magnitude for each C1 for each pixel provides direct roundtrip TOF data to the object point upon which such pixel is focused, e.g.: i1=C1·(ΔV/Δt), where i1 is current output by a pixel detector, C1 is the associated current integrator, ΔV is the signal change across C1 resulting from accumulated charge current, and Δt is the time over which the charge is accumulated.
Preferably one set of data is collected with all shutters remaining open for perhaps the period of the emitted light pulse train frequency. Charge gathered during this data set represents point-by-point reflected luminosity for the object surface. Such charge permits correcting for errors caused by more distant but more reflective object surface portion. For TOF>0, the underlying relationship is given by:TOF=[(Vcs−Vos)/Vcs]·PW+Ts−Tep,  Eq. (5)where Vcs is the signal resulting from the continuously open shutter, Vos is the signal resulting from a shutter opened for substantially the light pulse width duration, Tep is time of pulse end, Ts is time shutter closed, and PW is the time duration of the light pulse width.
If a shutter remained open, the total number of photons falling on a pixel detector is given by No=K·PW, where K is a constant that depends on the illumination intensity of the object, object reflectivity, and lens aperture. If Ts−(t0+2Z/C)<PW, the shutter will clip the tail-end of the reflected light pulse energy, and the number of effective photons received by the pixel detector will become:Ns=K·(Ts−t0−2Z/C)  Eq. (6)from which distance Z is obtained as:Z=C/2·(Ns·PW/No+t0−TS)  Eq. (7)
Accuracy of the above method depends on how accurately Ns can be measured. Ns and No are obtained by measuring the resulting charge on the sensor, for example using an A/D converter. Using a laser light source power of about 10 W, a 1 cm distance difference can result in variation ΔNs of Ns of about a hundred photons. To boost the accuracy multiple pulses in a cumulative fashion can be performed. Optical source 220 is pulsed repeatedly without clearing charge on the sensor between pulses. The charge on the sensor is thus the cumulative charge produced by all the reflected light pulses for which photo energy is detected. The cumulative charge is proportional to the cumulative number of effective photons summed over the number (n) of all pulses. By way of example, charge difference for a 1 cm distance change will be n·ΔNs, which for n=100 results a charge difference n·ΔNs on the order of tens of thousands of photons.
It will be appreciated that on one hand system 200 may produce inadequate results if incoming optical energy 35′ is too low in magnitude to be detected accurately. On the other hand, if the magnitude of incoming optical energy 35′ is too large in magnitude, various pixel detectors 240 may saturate, which also would preclude accurate detection measurements.
It is apparent from the foregoing that accurate TOF measurements depend upon the ability of sensor array 230 to output meaningful detection data over a wide magnitude of incoming optical energy 35′. Thus there is a need for a system and method to enhance dynamic range and to improve weak signal detection from TOF sensors.
The present invention provides such a system and methods.